Project export with 3d tasks #3502

ActiveChooN · 2021-08-03T10:57:44Z

Motivation and context

Added ability to export project dataset with 3d tasks
Depends on #3365

How has this been tested?

Manually

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
~~- [ ] I have updated the documentation accordingly~~
~~- [ ] I have added tests to cover my changes~~
~~- [ ] I have linked related issues (read github docs)~~
- [ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

Co-authored-by: Maxim Zhiltsov <maxim.zhiltsov@intel.com>

…t/cvat into dk/project-export

cvat/apps/dataset_manager/bindings.py

zhiltsov-max · 2021-08-06T15:33:02Z

cvat/apps/dataset_manager/bindings.py

@@ -940,9 +924,11 @@ def map_label(name): return label_cat.find(name)[0]
        return convert_cvat_anno_to_dm(cvat_frame_anno, label_attrs, map_label, self._format_type, self._dimension)

 class CVATProjectDataExtractor(datumaro.Extractor, CVATDataExtractorMixin):


Think of implementing this class as (roughly):

task_datasets = { task_id: make_task_dataset(task) for task_id, task in tasks } # append name mangling and other transforms like for task_id, task_dataset in task_datasets.items(): task_dataset.transform('rename', ...) dataset = Dataset.from_extractors(*task_datasets)

bsekachev · 2021-08-12T06:57:34Z

I am not expert in cvat/apps/dataset_manager, so I only tried this patch and was able to export both tasks and projects in 3D format. Just one question, if we dump 3D tasks we can only choice between PCD and kitty formats. But when we dump a project with one 3D tasks it shows all the formats (at the same time I cannot create a project with both 3D and 2D data). Is this expected behaviour?

bsekachev · 2021-08-12T07:22:06Z

@zhiltsov-max Do you have any new comments on the PR? I didn't see your approve

ActiveChooN · 2021-08-12T09:18:25Z

But when we dump a project with one 3D tasks it shows all the formats (at the same time I cannot create a project with both 3D and 2D data). Is this expected behaviour?

@bsekachev, fixed

zhiltsov-max · 2021-08-16T08:02:53Z

cvat/apps/dataset_manager/bindings.py

                        attributes["track_id"] = -1
-                        index += 1

                dm_item = datumaro.DatasetItem(id=osp.split(frame_data.name)[-1].split('.')[0],


Fix to my reported problem 3: Use either os.path.splitext() or str.rsplit(s, maxsplit=1)

Use of split() here does not conform to 2d formats and established CVAT behavior here, because 2d formats preserve the relative path information (to handle the problem with frames with same names in different dirs). Yes, it was discussed in the original 3d PR, but they refused to change it.

zhiltsov-max · 2021-08-16T08:03:55Z

My comments:

I think, it would be more user-friendly, if delete had red background and was positioned after exporting. How about renaming "export task dataset" to "export (as) dataset"? Two very similar names confuse.
2.
After the button is pressed and the menu disappears, when I reopen the menu, it has the button highlighted.
3. Export in sly pcd of a task with images:

и.м.г@ 1.JPEG
им. 536.45.--+=.г 3.jpeg
им.png.г 2.jpg
имг 3 (копия 1).jpeg

Error: Request failed with status code 500. "datumaro.components.errors.RepeatedItemError: Item ('им', 'qqqaaq') is repeated in the source sequence.\n".

2d project dataset export menu has 3d formats. Task export doesn't
The display of subset names looks a bit unclear to me. Maybe, it should be improved.
I didn't see 2d / 3d task / project indicator anywhere. Maybe, it's worth adding.

ActiveChooN · 2021-08-17T06:46:54Z

@zhiltsov-max

I think, it would be more user-friendly, if delete had a red background and was positioned after exporting.

Fixed. I'm not sure about the red color on menu items, there is already a red button on the delete modal.

Export in sly pcd of a task with images:

Fixed

Other comments are not very related to this PR so I moved them to a separate issue.

ActiveChooN · 2021-08-17T12:26:59Z

@dvkruchinin, hi, can you add tests for this feature, please?

dvkruchinin · 2021-08-17T12:30:04Z

@dvkruchinin, hi, can you add tests for this feature, please?

Sure. A test will be prepared.

ActiveChooN and others added 30 commits June 22, 2021 10:13

init

c53e062

Fixed typos

46e89b0

Merge branch 'develop' into dk/project-export

22fafe3

Fixed cache time value

7f4a349

Added MOTS PNG

4a5bc3a

Merge branch 'develop' into dk/project-export

c57ab56

temp

04d14c5

Another temp

f6b833f

WIP on UI

d45bec8

Added working modal

b23a9ba

Moved task export to modal

4c22e11

Fixed vscode files

a4d6094

Removed task actions

aadc83c

Fixed image_maker calling

bcd2034

Separated DataExtractor class

3aa16a1

wip

b59ab03

Added export with CVAT format

5d9b512

Fixed format resetting

a914d05

Fixed dir creating

3b480fa

Fixed export in CVAT formats

2647eb9

Added simple server tests, fixed datumaro format export

15d226a

Reverted settings change

3da98c4

Merge branch 'develop' into dk/project-export

d70ee77

Fixed merge

b8b3d9c

Fixed some comments

e1f8db6

Update cvat/apps/dataset_manager/bindings.py

98c9083

Co-authored-by: Maxim Zhiltsov <maxim.zhiltsov@intel.com>

Fixed comment

84f8b7a

Merge branch 'dk/project-export' of https://github.com/openvinotoolki…

4f8d28d

…t/cvat into dk/project-export

Added restrictions for 3d tasks

29ffac1

Fixed validator

4e2aac1

zhiltsov-max reviewed Aug 6, 2021

View reviewed changes

cvat/apps/dataset_manager/bindings.py Show resolved Hide resolved

zhiltsov-max reviewed Aug 6, 2021

View reviewed changes

ActiveChooN added 3 commits August 9, 2021 14:01

separated user info function

ddf8c3b

Fixed subset missing

bed869c

Added prefetch

d2c03c9

ActiveChooN mentioned this pull request Aug 10, 2021

Refactor project export using datumaro.Tranform #3535

Closed

2 tasks

ActiveChooN requested a review from zhiltsov-max August 10, 2021 09:53

ActiveChooN added 2 commits August 10, 2021 13:24

Added CHANGELOG

88a3ace

Fixed CHANGELOG

f1cb156

bsekachev self-assigned this Aug 11, 2021

bsekachev previously approved these changes Aug 12, 2021

View reviewed changes

Added dimension to project

ac2790f

ActiveChooN dismissed bsekachev’s stale review via ac2790f August 12, 2021 09:17

Fixed serilazer getting

84f328c

bsekachev previously approved these changes Aug 16, 2021

View reviewed changes

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

Fixed comments

579d2b4

ActiveChooN dismissed bsekachev’s stale review via 579d2b4 August 17, 2021 06:28

ActiveChooN mentioned this pull request Aug 17, 2021

Project UI improvement #3556

Closed

2 tasks

zhiltsov-max approved these changes Aug 17, 2021

View reviewed changes

nmanovic approved these changes Aug 17, 2021

View reviewed changes

nmanovic merged commit cef42b6 into develop Aug 17, 2021

nmanovic deleted the dk/project-export-3d branch August 17, 2021 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Project export with 3d tasks #3502

Project export with 3d tasks #3502

ActiveChooN commented Aug 3, 2021 •

edited

Loading

zhiltsov-max Aug 6, 2021 •

edited

Loading

bsekachev commented Aug 12, 2021

bsekachev commented Aug 12, 2021

ActiveChooN commented Aug 12, 2021

zhiltsov-max Aug 16, 2021 •

edited

Loading

zhiltsov-max commented Aug 16, 2021 •

edited

Loading

ActiveChooN commented Aug 17, 2021

ActiveChooN commented Aug 17, 2021

dvkruchinin commented Aug 17, 2021

		@@ -940,9 +924,11 @@ def map_label(name): return label_cat.find(name)[0]
		return convert_cvat_anno_to_dm(cvat_frame_anno, label_attrs, map_label, self._format_type, self._dimension)

		class CVATProjectDataExtractor(datumaro.Extractor, CVATDataExtractorMixin):

Project export with 3d tasks #3502

Project export with 3d tasks #3502

Conversation

ActiveChooN commented Aug 3, 2021 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

zhiltsov-max Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

bsekachev commented Aug 12, 2021

bsekachev commented Aug 12, 2021

ActiveChooN commented Aug 12, 2021

zhiltsov-max Aug 16, 2021 • edited Loading

Choose a reason for hiding this comment

zhiltsov-max commented Aug 16, 2021 • edited Loading

ActiveChooN commented Aug 17, 2021

ActiveChooN commented Aug 17, 2021

dvkruchinin commented Aug 17, 2021

ActiveChooN commented Aug 3, 2021 •

edited

Loading

zhiltsov-max Aug 6, 2021 •

edited

Loading

zhiltsov-max Aug 16, 2021 •

edited

Loading

zhiltsov-max commented Aug 16, 2021 •

edited

Loading